• Exploratory data analysis (EDA) covers roughly the "transform" and "visualize" steps.

• Ask an impactful question of a real data set.
• Use EDA to propose candidate answers.
• Create a simple business intelligence (BI) dashboard to guide decision-makers.

• You can do EDA in "base" R without any packages.
• But base R is a bad programming language.
• We will use the Tidyverse, a set of packages that promote code which is easy to write and read, highly performant, and consistent through the data scientific pipeline.
• The Tidyverse has been extensively developed by Hadley Wickham and collaborators over the last decade.

If you have prior experience in R and did not begin all your scripts with library(dplyr)….

• x %>% f() \(\Longleftrightarrow\) f(x)
• "Take x, and then do f to it"
• x %>% f(y) \(\Longleftrightarrow\) f(x,y)
• x %>% f(y) %>% g(z) \(\Longleftrightarrow\) g(f(x,y),z)
• "Take x, then do f with option y, then do g with option z…"

# familiar
listings %>% glimpse()  # = glimpse(listings)
listings %>% head()     # = head(listings)
listings %>% colnames() # = colnames(listings)

# get all columns with "review_scores" in the column name
listings %>% select(contains('review_scores')) 
# what should this return? 
listings %>% select(contains('review_scores')) %>% colnames()
# compare: colnames(select(listings, contains('review_scores')))

Let's try this out – back to the case study!

• You should usually summarise your data before turning on the fancy algorithms – sometimes the story is clear.

• data %>% mutate(new_col = formula(old_col1, old_col2) creates new columns.
• data %>% group_by(col) groups data for breakout summaries.
• data %>% summarise(measure = formula(col1, col2)) computes summaries.
• data %>% group_by(col) %>% summarise(measure = formula(col1, col2)) computes breakout summaries.
• Let's test these out in the case study.

calendar %>% 
    summarise(earliest = min(date), 
              latest = max(date))

## # A tibble: 1 x 2
##   earliest   latest    
##   <date>     <date>    
## 1 2016-09-06 2017-09-05

But some of these listings may be "zombies" without recent availability. How can we include only listings with availability from a certain time period?

1. Operate on the calendar table (exercise)
2. join that information to the listings table (together)
3. Filter the listings table accordingly (together)

The information we need is distributed between two tables – how can we get there?

We need a key column that tells us which calendar rows correspond to which listings.

listings$id corresponds to calendar_listing$id

The join family of functions lets us add columns from one table to another using a key.

• x %>% left_join(y) : most common, keeps all rows of x but not necessarily y.
• x %>% right_join(y) : keeps all rows of y but not necessarily x.
  
• x %>% outer_join(y) : keeps all rows of both x and y
• x %>% full_join(y) : keeps only rows of x that match in y and vice versa.

We'll use left_join for this case. On to the exercise!

Graphical excellence is the well-designed presentation of interesting data – a matter of substance, of statistics, and of design. Graphical excellence consists of complex ideas communicated with clarity, precision, and efficiency.

– Edward Tufte

A grammar is a set of components (ingredients) that you can combine to create new things. Many grammars have required components: if you're missing one, you're doing it wrong. In baking….

• Puts the gg in ggplot2.
• Formulated by Leland Wilkinson.
• Implemented in code by Hadley Wickham, now part of the tidyverse

• Data: almost always a data_frame
• Aesthetic mapping: relation of data to chart components.
• Geometry: specific visualization type? E.g. line, bar, heatmap?
• Statistical transformation: how should the data be transformed or aggregated before visualizing?
• Theme: how should the non-data parts of the plot look?
• Misc. other options.
• (+ plays the same role in ggplot2 that %>% does in data manipulation.)

The following code computes the average price of all listings on each day in the data set:

average_price_table <- calendar %>% 
    mutate(price = price %>% gsub('\\$|,', '',.) %>% as.numeric()) %>% 
    group_by(date) %>% 
    summarise(mean_price = mean(price, na.rm = TRUE))

Use geom_line() to visualize these prices with time on the x-axis and price on the y-axis.

 average_price_table %>% 
    ggplot() + 
    aes(x = date, y = mean_price) + 
    geom_line()

Using the summary_table object you created earlier, make a bar chart showing the number of apartments by neighbourhood. In this case, the correct geom to use is geom_bar(stat = 'identity').

• Working with your partner, you are going to make a business intelligence (BI) dashboard for AirBnB, using the EDA skills we have developed in this session.
• You will use this dashboard to lead a meeting with decision-makers on where to prioritize host recruitment efforts.
• It will look like this – but better!

• Data Wrangling Cheatsheet
• ggplot2 Cheatsheet
• The R Graphics Cookbook, by Winston Chang

• Jenny Bryan's fabulous course website on introductory, reproducible data analysis with R
• R for Data Science, a book on the Tidyverse by Hadley Wickham and Garrett Grolemund
• Modules 2-4 of "Computing in Optimization and Statistics" (15.S60) covers similar material in more breadth and depth.

• Reproducible research with RMarkdown
• Interactive data applications with Shiny
• Automate long data science pipelines with GNU make
• Develop your own R packages
• Some news on R, statistics, and data science at FiveThirtyEight (they use R!)